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Abstract 

In this paper we consider a family of optimal control problems for 
economic models whose state variables are driven by Delay Differential 
Equations (DDE's). We consider two main examples: an AK model with 
vintage capital and an advertising model with delay effect. These prob- 
lems are very difficult to treat for three main reasons: the presence of the 
DDE's, that makes them infinite dimensional; the presence of state con- 
straints; the presence of delay in the control. Our main goal is to develop, 
at a first stage, the Dynamic Programming approach for this family of 
problems. The Dynamic Programming approach has been already used 
for similar problems in cases when it is possible to write explicitly the value 
function V (see Here we deal with cases when the explicit form of 

V cannot be found, as most often occurs. We carefully describe the basic 
setting and give some first results on the solution of the Hamilton- Jacobi- 
Bellman (HJB) equation as a first step to find optimal strategies in closed 
loop form. 



1 Introduction 

In this paper we want to develop the Dynamic Programming approach for a 
family of optimal control problems related to economic models governed by 
Delay Differential Equations (DDE's). 

The presence of DDE's makes the problem difficult to treat. One possible 
way of dealing with DDE's - the one we choose - is rewriting the problem as 
an optimal control problem governed by ODE's in a suitable Hilbert space. 
Although such infinite dimensional optimal control problems have already been 
studied, the present literature does not cover our case, as it does not include 
the following features: 

• the presence of unbounded operators coming from the DDE which is not 
analytic and does not satisfy smoothing assumptions; 

• the presence of state / control constraints (which is indeed peculiar of eco- 
nomic models); 
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• the fact that the delay appears in the state and in the control (causing 
the control operator to be possibly unbounded). 

We stress the fact that these difficulties are the rule in economic models governed 
by DDE's. 

Here we consider problems with linear DDE's and concave objective func- 
tional: concavity will play a key role in the paper. When concavity lacks, one 
can still apply Dynamic Programming in the framework of viscosity solutions - 
which we avoid here. Nevertheless, we address the reader to |8!] for a standard 
reference on viscosity solutions. 

We remark that this is a first step in treating such kind of problems. We 
already studied thoroughly in pj] a case where explicit solution of the associ- 
ated Hamilton-Jacobi-Bellman (H.JB) equation can be found (in such case the 
problem is much easier to treat). Here we want to develop the Dynamic Pro- 
gramming approach in those cases when explicit solutions of the associated H JB 
equation are not available. We here develop the finite horizon case. The infinite 
horizon case can be treated with our method using a Hmiting procedure when 
the horizon goes to -|-oo but we leave it for future work^. 

The main result of the paper is that the value function of the problem is a 
solution, in a suitable weak sense, of the HJB equation. This a first step towards 
the so-called Verification Theorem which is a powerful tool to study the optimal 
paths of the problem and which is the subject of our current research. 

We concentrate on two main examples: an AK model with vintage capital, 
taken from ^ (see also jH] and advertising model with delay effects 

(see |24ll25j l that are exposed in Section [21 

The plan of the paper is the following. In Section 12 we present the applied 
examples. In Section El we recall the basic steps of the Dynamic Programming 
approach and we give an overview of the current literature on the Dynamic 
Programming for infinite dimensional optimal control problems. In Section 

01 we rewrite the state equation of such problems as an ODE in a suitable 
Hilbert space, concentrating on the first example, as the second can be rephrased 
similarly. In SectionOwe write the resulting infinite dimensional optimal control 
problem and its H.IB equation. Sectionj^lwe show our main result: the existence 
of an ultraweak solution of the HJB equation. The Appendix contains some 
definition and proof that may be useful for the reader. 

2 Two examples 

We present the two applied problems motivating this paper. 
2.1 An AK model with vintage capital 

We consider here an optimal control problem related to a generalization of the 
model presented by Boucekkine, Puch, Licandro and Del Rio in j^. Indeed, we 
assume that the system is ruled by the same evolution law as the one in [Jj, that 
is, an AK growth model with a stratification on the capital. Besides, we consider 
the finite horizon problem with a (more) general concave target functional, as 

^In thus respect we can say that the finite horizon case is as a first step towards the infinite 
horizon one. 
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specified later. The analysis of such a model proves interesting in the study of 
short run fluctuations and of transitional dynamics: the reader is referred to 
[£| for a deep discussion upon this and other related matters. The model of j2] 
is an infinite horizon model, while here we consider the finite horizon case. As 
mentioned in the introduction, this is a first step towards the infinite horizon 
case. 

The AK-growth model with vintage capital is based on the following accu- 
mulation law for capital goods 

fc(s) — I i((T)d(T 
J s-R 

where z(cr) is the investment at time a. That is, capital goods are accumulated 
for the length of time R (scrapping time) and then dismissed. Note that such 
an approach introduces a differentiation in investments that depends on their 
age. If we assume a Hnear production function, that is 

y{s) = ak{s) 

where y{s) is the output at time s (note that "^i^" reminds of the Hnear 
dependence of the dynamic from the trajectory - a constant A multiplied by K\ 
such constant A is a in our case), and we assume also the accounting relation 

y{s) = c(s) + i(s), 

meaning that at every time the social planner chooses how to split the produc- 
tion into consumption c(s) and investment i(s), then the state equation may be 
written into infinitesimal terms as follows 

fc(s) = afc(s) - ak{s - R) - c{s) + c{s - i?), se[t, T] 

i.e. as a DDE. The time variable s varies in [t,T], with t the initial time and T 
the (finite) horizon of the problem. Indeed, the social planner has to maximize 
the following functional 

j\-P'ho{c{s))ds + MHT)) (1) 

where ho and 0o are concave u.s.c. utility functions. We recall that in the 
horizon is infinite and (po — 0. Moreover the instantaneous utility is CRRA (i.e. 
Costant Relative Risk Aversion), that is the function ho is of type ho{c) = j:^, 
which satisfies our assumptions as a subcase. 

Observe that we take the starting time t to be variable to apply the finite 
horizon dynamic programming. 

We assume that the capital at time s (and consequently the production) and 
the consumption at time s cannot be negative: 

k{s)>0, c(s)>0, Vse[t,r] (2) 

These constraints are different from the more restrictive and more natural ones 
of (3, where also the investment path was assumed positive. 

The main reason for such a choice is technical: we cannot apply the strong 
solution approach that we use in this work with mixed constraints such as 
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those in The treatment of mixed constraints is also left for future work. 
We mention indeed that the optimal solutions for the problem without mixed 
constraints may satisfy in some cases the positivity of investments, yielding the 
solution also for the problem with mixed constraints. 

In order to take the constraints into account, we assume that the consump- 
tion (that is, the control variable of the system) lies in the following admissible 
set 

A'^= {c{-) e L'^{[t,T],R) : c{-)>Oandk{-)>Oa.e.in[t,T]}. 

2.2 An advertising model with delay efFects 

Another example of optimal control problems driven by DDE's is the following 
a dynamic advertising model presented in the stochastic case in the papers 
[211 EI, and, in deterministic one, in [2l] (see also [?] and the references therein 
for related models)^. 

Let i > be an initial time, and T > t a terminal time (T < +oo here). 
Moreover let 7(s), with < t < s < T, represent the stock of advertising 
goodwill of the product to be launched. Then the general model for the dynamics 
is given by the following controlled Delay Differential Equation (DDE) with 
delay R> where z models the intensity of advertising spending: 

r 7(s) = ao7(s)+/°^7(s + e)rfai(e)+&o^(s) + /°^^(s + e)d6i(0 s € [t,T] 

I lit) = X- 7(6 = 0{i\ <0 = SiO ve e [t - i?, t], 

(3) 

with the following assumptions: 

• flo is a constant factor of image deterioration in absence of advertising, 
ao < 0; 

• ai(-) is the distribution of the forgetting time, ai(-) G L^([— i?, 0]; K); 

• 6o is a constant advertising effectiveness factor, bo > 0; 

• 6i(-) is the density function of the time lag between the advertising ex- 
penditure z and the corresponding effect on the goodwill level, 6i(-) G 

L\[-R,0];R+); 

• X is the level of goodwill at the beginning of the advertising campaign, 

x>0; 

• 9{-) and 6{-) are respectively the goodwill and the spending rate before 
the beginning, e{-) > 0, with 61(0) = x, and S{-) > 0. 

Note that when ai(-), bi{-) are identically zero, equation (pj reduces to the 
classical model contained in the paper by Nerlove and Arrow (1962). We assume 
that the goodwill and the investment in advertising at each time s cannot be 
negative: 

7(s)>0, z(s)>0, Vse[t,r]. (4) 

^We observe that also other models of delay type arising in economic theory can be treated 
with our tools (see e.g. the paper by 
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Finally, we define the objective functional as 

J{t,x;z{-))^Ml{T))- ho{z{s))ds, (5) 

where tpo is a concave utility function, ho is a convex cost function, and the 
dynamic of 7 is determined by (EJ . The functional J has to be maximized over 
some set of admissible controls Z//, for instance L^([i, T]; R+), the space of square 
integrable nonnegative functions. 

3 The dynamic programming approach 

The Dynamic Programming (DP) approach to optimal control problems can be 
summarized in four main steps (see for instance Fleming and Rishel [SI for the 
DP in the finite dimensional case and Li and Yong [221 for the DP in the infinite 
dimensional case): 

(i) letting the initial data vary, calling value function the supremum of the ob- 
jective functional and writing an equation whose candidate solution is the 
value function: the so-called DP Principle, together with its infinitesimal 
version, the Hamilton- Jacobi-Bellman (HJB) equation; 

(ii) solving (whenever possible) the HJB equation to find the value function; 

(iii) proving that the present value of the optimal control strategy can be 
expressed as a function of the present value of the optimal state trajectory: 
a so-called closed loop (or feedback) relation for the optimal control; 

(iv) solving, if possible, the Closed Loop Equation (CLE), i.e. the state equa- 
tion where the control is replaced by the closed loop relation: the solution 
is the optimal state trajectory and the optimal control strategy is conse- 
quently derived from the closed loop relation. 

Such method, when applicable, allows one to give a powerful description of 
the optimal paths of an optimal control problem. 

First of all we clarify that the two models above are not easy to manage with 
the DP approach as they presents two special difficulties. 

• The state equation is a Delay Differential Equation while the DP ap- 
proach is generally formulated for controlled Ordinary Differential Equa- 
tion (ODE). One way to approach the issue (for a different one, see e.g. 
Kolmanowskii and Shaikhet ^3) is to rewrite the DDE as an ODE in an 
infinite dimensional space, which plays the role of the state space. We 
use in the sequel the techniques developed by Delfour, Vinter and Kwong 
(see Section 31 below for explanation and Subsection l.S.ll for references). It 
must be noted that the resulting infinite dimensional control problem is 
harder than the ones usually treated in the literature (see e.g. [22]) due 
to the unboundedness of the control operator and the non-analyticity of 
the semigroup involved (see again Subsection^. 

• Both problem feature pointwise constraints on the state variable, see l|2l, 
Q. Their presence makes the problem much more difficult, and only a few 
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results in special cases (different from the one treated here) are available 
in the literature. Indeed for such problems in infinite dimension there 
is no well established theory. This fact is at the basis of the theoretical 
problem contained in the paper [Jj and mentioned in pj] point (II) in 
the introduction: show that the candidate optimal trajectory satisfies the 
pointwise constraints (EJ. 

To overcome such difficulties in |T2] we show that for our special problem we 
can exhibit an explicit solution of HJB equation. This is the key result that 
allows to complete the DP approach in |17j . 

Here, since we do not want to write the utility functions in a fixed explicit 
form (Hke the CRRA used in [3 EI), we cannot obtain an an expHcit solution 
of HJB equation. Therefore we would like (here and in the future) to perform 
the following steps: proving existence (and possibly, uniqueness) for the HJB 
equation, then some theoretical results of type (iii) and (iv) above, and hopefully 
some subsequent numerical approximation. This is a wide and difficult program. 
In this paper we take just a first step towards the scope: existence results for 
the HJB equation. 

3.1 The literature on Delay DifFerential Equations and on 
Dynamic Programming in infinite dimensions 

For Delay Differential Equations a recent, interesting and accurate reference is 
the book by Diekmann, van Gils, Verduyn, Lunel and Walther |TQ. 

The idea of writing delay system using a Hilbert space setting was first due 
to Delfour and Mitter Jij, J^I- Variants and improvements were proposed 
by Delfour [Til, 0, 1101, Vinter and Kwong [30,, Delfour and Manitius [12] . 
Ichikawa |2£] (see also the precise systematization of the argument in chapter 4 
of Bensoussan, Da Prato, Delfour and Mitter 0). 

The optimal control problem in the (linear) quadratic case is studied in 
Vinter, Kwong [SQ, Ichikawa [22|, Delfour, McCalla and Mitter In that 

case the Hamilton-Jacobi-Bellman reduces to the Riccati equation. 

The study of Hamilton-Jacobi-Bellman equation in Hilbert spaces, started 
with the papers of Barbu and Da Prato [21, [HI, is a large and diversified 
research field. We recall that the best one may achieve is a "classical" solution 
of HJB equations (i.e. solutions that are differentiable in time and state) since 
this allows to get a more handleable closed loop form of the optimal strategy. 
Since classical solutions are not always available, there is a second stream in 
the literature that studies the existence of "weak" solutions (i.e. solutions that 
are not differentiable)^. In this paper we investigate existence of a weak-type 
solution (that we call ultraweak, see Section ^ that are limits of classical solu- 
tions. Up to now, to our knowledge, the existence of such solutions for the HJB 
equation in cases where the state equation is a Delay Differential Equation has 
not been studied in the literature (apart from the Hnear quadratic case) . In the 
economic literature the study of infinite dimensional optimal control problems 
that deals with vintage/heterogeneous capital or advertising models is a quite 
recent tool but of growing interest: see for instance 0, |22], ^S], |24[l25j . 

^The most general concept of weak solution is the one of viscosity solution, introduced by 
Crandall and Lions in the finite dimensional case and then applied to infinite dimension by 
the same authors, see [8| for an introduction to the topic and further references. 
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4 The state equation in an infinite dimensional 
setting. 



In this section we show how to rewrite the state equations of our examples as 
controlled ODE's in a suitable Hilbert space. We do it thoroughly for the first 
example, as the second is similar and simpler. 

4.1 Notation and preliminary results 

In this section we recall some general results on delay differential equations 
(DDE) and on the related Hilbert space approach, as applied to our case. The 
reader is referred to the book by Bensoussan, Da Prato, Delfour and Mitter 
[D for details. We consider from now on fixed i? > 0, and a > 0. With 
notation similar to that of [5], given T > t > and z e L'^{[t - R,T],R) 
(or z € Lfocii^ ~ R:+oo),R)), for every s e [t,T] (or s e [t, +oo)) we call 
Zs e L2([-i?,0];M) the function 

f Zs : [-R, 0] ^ R 
I Zs{<7) z{s + a) 

Given a control c G ^ we consider the the following delay differential equa- 
tion: 

J k{s) = ak{s) — ak{s — i?) — c(s) + c{s — R) for s G [t, T] , s 

{ {k{t),kt,ct) ^ {(f>",(f>\uj) eRx L^{[-R,0];R) X L^i[-R,0];R) 

where kt and Ct are interpreted by means of the definition above. Note that in 
the delay setting the initial data are a triple, whose first component is the state, 
the second and third are respectively the history of the state and the history of 
the control up to time t (more precisely, on the interval [t — R,t]). The equation 
does not make sense pointwise, but has to be regarded in integral sense. We 
give now a more precise existence result and an estimate on the solution: 

Theorem 4.1. Given an initial condition {(f)^ , (j)^ , oj) e M x L^([-_R, 0]; M) x 
i^([— i?, 0]; M) and a control c e L^([i,T],R) there exists a unique solution fc(-) 
of ^ m T4^^'^([i, T], M). Moreover there exists a positive constant C {T — t) such 
that 

|fc|lVi.2([t,T],K) < C{T - t){^cfP\ + |(?I)^|L2([_fl_o];R) + |w|L2([-i?,,0];R) + \c\L^{[t.T],K-)) 

(7) 

Proof. See [1] Theorem 3.3, p.217 for the first part and Theorem 3.3 p. 217, 
Theorem 4.1 p. 222 and p. 255 for the second statement. □ 

In view of the continuous embedding W'^^'^{[t, r],M) C°{[t,T],M.) we have 
also: 

Corollary 4.2. There exists a positive constant (possibly different from the one 
above) C{T — t) such that 

|fc|cO([t,T],R) < C'(r-t)(|(?!)°| + |(/)^|L2([-fl^0];R) + l^lL2([-_R,0]:R) + |c|L2([tj.]j{) j (8) 
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We consider now the continuous linear application L with norm ||L|| 

L: C([-i?,0],M) ^ M 

and then define £* as follows 

where £*(</>) : s i-> L{(j),) for s e [t, T] ^ ' 

where Cc{t — R,T;R) is the set of real continuous functions having compact 
support contained in {t — R, T) 

Theorem 4.3. The linear operator £* : Cc{[t - R, T], M) L'^{[t, T], R) has a 
continuous extension C* : L'^{[t — R,T],M.) L^([i,T],M) with norm < \\L\\ . 

Proof. See Theorem 3.3, p. 217. □ 

Using the "L" notation we can rewrite (EJ as 

k{s) = aL{ks) - L{c,) for s £ [i, T] 

{k{t),kt,ct) = (<?!)°,0\t^) e M X L2([-i?,0];R) x L'^{[-R,Q\:R) 
and using the notation we can rewrite ^ as 
k{s)^a{C'k){s)^[C'c){s) forse[t,T] 

{k{t),kt,ct)^i<l)°,(j)\u;)eM.xL^{[-R,0];R)xL^{[-R,0];M.) ^ ' 

There follows another step towards the setting in infinite dimension that we 
intend to use. So far, the history of the control and of the trajectory were kept 
separated. Indeed one may note that the delay system depends jointly on those 
data. Such joint dependence is exploited in the sequel to reduce the dimension 
of the state space. We then need to add some more notation to make this more 
explicit. 

• Given u € L'^{[t - R,T],R) we define the function e*+M e L'^{[t - R,T],R) 
as follows 



e* u: [t - i?,T] R, e*u{s) 



u{s) se[t,T] 
s e [t- R, t) 



Given u G L'^{[-R,{)\]R) we define the function e"_w £ L'^{[t - i?,r],] 
as follows 



s e [t, T] 

u{s -t) se[t- R,t) 



Given a function u e i?, 0]; R) and s E [<, T] we define the function 

r]{s)u e L^{[~R,0];R) as follows 



7j{s)u: [-R,0]^R, {7j{s)u){e) = 



u{-s + t + e) e>-R + s-t 
e < -R + s-t 
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Note that k = e*^fc + 6*10^, and c = e'^_^c + e'^_uj , then we can separate the 
solution k{s), s > t and the control c(s), s >t from initial data 0^ and u: 

k = aC^e\k - L^e\c + a/:*eO <?ii - £*eO w , . 

k(t) = 00 e M ^ > 

Note that system lfTT|l does not directly use the initial function </)^ and but 
only the sum of their images a£*e^0^ — C}e^_LO. We need a last step before we 
can write the delay equation in Hilbert space. We introduce the operator 

r L:L2([_i?,o];K)^i2([_^^0];M) 

\ (L<^i)(a) =^L(est(0i)_„)) ae(-i?,0) 

where est{(j)^) is the function R ^ M that achieves value out of {—R,0) and 
that is equal to (j)^ in {—R, 0) (the same for ui). 

Note that the operator L is continuous (see page 235), moreover 

a£*e°(/)i(s) -/:*e°w(s) = (r;(s)(aL0i - Lu;))(0) for s > t. 

Therefore, if we set 

=^(al0i-M, =^ (/.o, (13) 

then we can rewrite and consequently ijHJ as 

fc(s) = (a£*eVfc)(s) - {£'e'i^c){s) + {ri{s)x^){Q) for s>t 
k(t) = e M 



(14) 



where M x L'^{[-R,0];R) 3 x ix°,x^), c e A. Note that dl is meaningful 
for all X e R X L^([-i?,0];R), also when is not of the form ifT^ll . So we have 
embedded the original system © in a family of systems of the form H14|l . 

4.2 The state equation of the AK model in the Hilbert 
setting 

We now work on the following Hilbert space 

=^R X L^{[-R, 0];R) 

where the scalar product between two elements — and ^ = is 

given by 

Next we consider the homogeneous system 

z{s) = {aC°z){s) 
(z(O),zo) = 0eM2 

and define the family of continuous linear transformations on 

J Sis) : AP ^ M2 

I S{s)(l)'^^^ {z{s),Zs). 
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Then {>5'(s)}s>o is a Co semigroup on whose generator is 

D{G) = e M2 : 01 e M^i^2(_^^Q) ^0 ^ 01^0)} 

G(0O,0i) = {aL4>\D(P^) 

where D(j)^ is the first derivative of (p^. A proof of this assertion can be found 
in 0, Chapter 4. 

Note that the second component (p^ of the elements oiD{G) is in C{[—R, 0], M) 
so, with a sHght abuse of notation, we can re-define L on D{G) in the following 
way 

L: D{G) ->]R 
L(0O,0i) = L(/.i 

Moreover, if D{G) is endowed with the graph norm, we denote with j the 
continuous inclusion D{G) ^ M^. Hence the operators G, and j are continuous 
from D{G) into and L is continuous from D{G) into R. We call G* , j* and 
L* their adjoints, and identify and R with their dual spaces, so that 

G*: ^ D{GY 
j* : M2 i:)(G)' 
L*:R^ D{G)' 

are linear continuous. 

Definition 4.4. T/ie structural state x{s) at time t > is defined by 

y{s) (2/°(5),y'(s)) {k{s),aL{e\k)s~L{e\c),+T^{s)x^) (15) 

In the sequel we use and to indicate respectively the first and the second 
component of the structural state. We can give also a different, more expHcit, 

definition: if we call fcs, c^e L^([— i?, 0]; M) the appHcations 

ks-d^ -fc(s -R-9) 
'cs-.O^ -c{s-R-0) 

the structural state can be written as 

y{s) {k{s),a ks - +v{s)x^). (16) 

Eventually, we write the delay equation in the Hilbert space by means 
of the following theorem. 

Theorem 4.5. Let y'^{s) be the solution of system for x e A'P , c E A and 
let y(t) be the structural defined in Then for each T > 0, the state y is the 

unique solution in 

feC{[t,T],M') ■■ ^.ff eL\[t,T],D{Gy) 
as 

to the following equation 

^y{s)^G*yis) + L*c{s) ^^^^ 
y{t) = X. 

Proof. See :5j Theorem 5.1 Chapter 4. □ 
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4.3 The state equation of the advertising model in the 
Hilbert setting 

Similar arguments can be used for the advertising model. We write here only 
the results. We call N, B the continuous linear functional given by 

N: C([-i?,0]) -^R 

B: C([-i?,0]) ^ R 

Let G be the generator of Co-semigroup defined as: 

D{G) = e AP : 01 G W^'^{-R,0) and 0° = 0i(O)} 

We define N and i? in the same way we defined L in equation Ijl2|l . So we can 
write the advertising model in infinite dimensional form. We obtain: 

• The structural state in the advertising model will have the following ex- 
pression: 

y{t) ^ {y^s),y\s)) {^{s) ,N {e\^) , -B{e\z) , ^{s)x^) 
where xi =N(ff)~^(b). 

• The state equation becomes 

^y(s) = G*2/(s) + i3*z(s) 
y{f) = X. 

5 The target functional and the HJB equation 

We now rewrite the profit functional for the first example in abstract terms, 
noting that a similar reformulation holds for the target functional of the sec- 
ond example. We consider a control system governed by the linear equation 
described in Theorem 14.51 We assume that the set of admissible controls is 
defined by 

y^''^ {c(-) ei2([t,T],]R) : c(-) > OaW2/0(-) > 0} 

As usual, the trajectory ?/(•) (and then ?/°(-)) depends on the choice of the 
control c(-), and of initial time and state, i.e. y(-) = y{-;t,x,c{-)), but we write 
it explicitly only when needed. 

In order to apply the results contained in (20] and recalled in the Appendix, 
we reformulate the maximization problem as a minimization problem. At the 
same time we take the constraints into account by modifying the target func- 
tional as follows. If ho and 0o are the concave u.s.c. functions appearing in 
then we define _ 

/i: R ^ R 

^ I -hoic) ^f c>0 
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Moreover we set 



-Mr) ifr>0 
+00 if r < 



g: K 

' if r>0 
+CO if r < 



Both h, (f) and g are convex l.s.c. functions on R. Then we define the target 
functional as 

At, ^, c(-)) = e-P^Hcis)) + g{y'{s))]ds + 0(y°(T)) 

with c varying in the set of admissible controls L^([t, T], M). It is easy to check 
that the problem of maximizing (QJ in the class A is equivalent to that of min- 
imizing J on the whole space L^([i,T],R). Then the original maximization 
problem for the AK-model has been reformulated as the following abstract min- 
imization problem: 

ini{J{t,x,c{-)) : ce L^{[t,T],R), andy naXisfieB (18) 

Moreover, HJB equation is naturally associated to such minimization problem 
by DP, and it is given by 

dtv{t, x) + {Vv{t, x), G*x) - F{t, Vv{t, x)) + e-P'g{x) = 
v{T,x)^Ux) ^"'"^ 

with F defined as follows 

J F: [0,T] X D{G) -> M 

\ F{t,p) =^ sup,>o {-Lip)c - e-P'hoic)} - e~P'h*{^eP'L{p)) 

where h* is the Legendre transform of the convex function h. We refer to F as 
to the Hamiltonian of the system''. 

The abstract framework is then set, and we are ready to perform Dynamic 
Programming. 



6 The value function as ultraweak solution of HJB 

We define the value function of the optimal control problem described in the 
previous sections as 

W{t,x)'^^ inf J(t,x,c{-)). 

c(-)GL2([t,T];R) 

Our objective here is to provide a suitable concept of solution of HJB, so that 
the value function is a solution, in such sense. 

^Note that, following the usual definition, the Hamiltonian should be indeed {p,G*x) — 
F{t,p) + e~P^g{x). Here, for commodity of notation, we put aside of the Hamiltonian the 
terms which are linear or constant in p. 
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We recall that in ^| it is shown that, if the data satisfy certain assumptions 
(involving convexity, semicontinuity, and coercivity of h), then the value func- 
tion of an optimal control problem with state constraints of type Ill8|l is indeed 



the unique weak solution to a HJB equation of type ijH.TBII . as there proved 
and here recalled in the Appendix, Theorem 17.111 Note that some coercivity 
for the function h is indeed lacking in our case, as the prototype of is ^^^^ 
as mentioned before, which is sublinear on the positive real axis. This causes 
the Hamiltonian of the problem - that is related to the Legendre transform of 
ho - to be possibly nonregular, so that all previous definition of solutions do 
not apply. (Note indeed that, as more precisely stated in the Appendix, a weak 
solution is limit of strong solutions of approximating equations, while a strong 
solution is itself limit of classical solutions of approximating equations. All of 
these notions require the Hamiltonian to be differentiable with respect to the 
co-state variable p.) 

Here we are about to define a uUraweak solution as limit of weak solutions 
to ljH.TB|l . Note that the concept of solution is indeed generalized, although 
not in the same direction as before, due to the presence of possibly nonregular 
Hamiltonians. 

According to the notation in ^^I) if ^ and Y are Banach spaces, we set 
L^p{X■, Y)^{f:X^Y : [/], := sup '^^f '^.^^^'^ < +00} 
Clp{X) ■.= {feC\X) : [/I <+oo} 

Cp{X,Y) ■.= {f:X^R : |/|c, := sup J^T^ < +^}' ^p(^) - CpiX,R). 
Moreover we set 

Eo(^) := {w e C2{X) : w is convex, w £ C]^^p{X)} 

y{[0,T]x X) = {w:[0,T]x X ; w e C{[0,T],C2{X)), 

wit,-) eJ^oiX), Vw G C{[0,T],Ci{X,X'))}. 

Definition 6.1. We say that a function V is a ultraweak solution to 

dtv{t, x) + {Vv{t, x),G*x) - F{t, \/v{t, x)) + e-P^g{x) = 
v{T,x) = (f>o{x) 

if there exists a sequence {Fn}n of functions in the space ^([0, T] x D{G)), such 
that Fn] F pointwise, and 

V{t,x)= lim Vnit,x) ^ inf Vn{t,x) 

+ n>0 

with Vn the unique weak solutions to 

dtv{t,x) + {Vv{t,x),G*x) - F^{t,\Iv{t,x)) + e~P^g{x) ^ 
v{T,x) = (f>o{x) 

Note that any weak solution V is convex in the state variable x, but not 
necessarily l.s.c in {t,x). We are able to prove an existence result for equation 
(|HJB|I by proving that the value function of the control problem set in the 
previous section is an ultraweak solution. 
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Theorem 6.2. The value function W of the optimal control problem USp is an 
ultraweak solution of fHJB\} . 

Proof. First of all we need to construct a sequence of Hamiltonians Fn having 
the properties required by the definition above. We choose 

F,,{t,p) ■.= e-P'hl{-eP'L{p)) 

with 

/i„(c) = h{c) + — |cp, n e N. 
2n 

Indeed if we denote with Snf{x) = inf^gH {/(y) + — j/p} the Yosida ap- 
proximation of a function /, then it is easy to check that [Snf]*(x) = f*(x) + 
^|a;p, so that 

hl{c) = S^{h*){c). 

Being /i* the Yosida approximations of a l.s.c. convex function, they result 
to be Frechet differentiable with Lipschitz gradient, with Lipschitz constant 
< n. Moreover, as /i„ is a decreasing sequence, Fn is then increasing, as 
required by Definiton l6.ll Hence the assumptions in Theorem 17. Ill are satisfied 
for the problem of minimizing the functional 

1 r 

Jnit,x,c)^J{t,x,c) + — / e-P'\c{s)\^ds 

in L'^{[t, r],K), and we easily derive as a consequence the following result. 
Lemma 6.3. Let 

Wn (t, x) inf Jn {t, x,c), 

ceL-^{[t,T]M) 

be the value functions of the approximating optimal control problem. Then Wn 
is convex in x and l.s.c. in x and t, and it is the unique weak solution of 

dtv{t, x) + {Vv{t, x),G*x) - F„(t, Vv{t, x)) + e-P^g{x) = 
v{T, x) = (j){x) 

Moreover there exists c* G L'^{[t, T], R) optimal for the approximating problems, 

i.e. Wn{t,x) ^ Jn{t,X,Cn). 

To complete the proof we need to show that Wn(t, x) J, W{t, x). 
Lemma 6.4. The value function of is given by 

W{t,x)^ lim Wn{t,x) ^mfWn{t,x). 

n — >oo n 

Proof. By definition of J„, for all t, x and n we have J„(i, x, c) > J„+i (t, x, c) 
for all admissible controls c, so that 

Wn{t,x) > Wn+l{t,x), 

and {Wn{t,x)}n is a decreasing sequence. As a consequence, an ultraweak 
solution V of HJB exists, and it is given by 

V{t,x) lim Wn{t,x) = inf Wn{t,x). 
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Next we show that a solution V built this way necessarily coincides with W. 
Note that 

J{t,x,c) < Jnit,x,c), \fc e L'^{[t,T],R), 
so that by taking the infimum and then passing to limits, we obtain 

W{t,x) <V{t,x). (19) 

We then prove the reverse inequality. Let e > be arbitrarily fixed, and be 
an e-optimal control for the problem, that is W{t, a;) + £ > J{t, x, Cg). Note that, 
by passing to limits as n ^ +00 in 

V{t,x) < Wn{t,x) < Jn{t,X,Ce) 

one obtains 

V{t, x) < J{t, X, Ce) < W{t, x) + e, 

which impHes, together with ifTflji . the thesis. 

Doing so we proved the lemma and Theorem 16. 21 

Remark 6.5. Note that we do not derive any uniqueness result for ultraweak 
solutions. If for instance one tries to get uniqueness by showing that any ultra- 
weak solution of HJB is the value function of a certain control problem, some 
difficulties arise, due to the fact that, although h*^] Ti if and only if there exists 
some h such that hn [ h, in general Ti* ^ h unless some minimax condition is 
satisfied, such as 

h = inf sup{cr — h*^{r)} — supinfjcr — h*^{r)} — Ti* , 
which is false in general. 



7 Appendix 

In this section we recall the abstract framework and the main results contained 
in and (20], regarding strong and weak solutions of HJB. 

In and [201 we worked in an abstract setting on some state space denoted 
with V . In that setting, if is a separable Hilbert space, Aq is the generator 
of a strongly continuous semigroup of operators on H, and V is the Hilbert 
space D{Aq) endowed with the scalar product {v\w)v ■= {v\w) h + {Aqv\Aqw) h , 
then we set V equal to its dual space endowed with the operator norm. The 
semigroup generated by ^0 can be extended in a standard way to a semigroup 
{e^''}s>o on the space V , with generator A, a proper extension of Aq. 

Then we assume the state equation in V is given by 

fy'{s)^Ayis) + Bc{s), s e [t,T] 

with control operator B e L{U,V') (although B ^ L{U,H)), where U is the 
control space and c G L'^{[t,T],U) the control. Such equation may be readily 
expressed in mild form as 

y{s) = e^^'-''^x+ [ e^^'-"'^ Bc{a)da. (21) 
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Remark 7.1. The role of V in the case of the delay equation here presented 
is played by the space D{G)' , and the role of by the operator G* . 

Besides, we consider a target functional Jo, associated to the state equation, of 
type ^ 

J{t,x,c)=J^ [g{s,y{s)) + h{s,c{s))]ds + ^{y{T)) (22) 

with h(t,-) real, convex, l.s.c., coercive, and g(t,-) and v real, convex, and 
C^{V') (respectively, l.s.c. in V) in the x variable, as more precisely stated in 
the next sections. The problem is that of minimizing J{t, a;, •) over the set of 
admissible controls L'^{[t,T];U). 

Remark 7.2. Indeed, in the applications, the target functional is rather of type 

Mt, c) = ^ K (s, + V {s, c{s))] ds + v{y{T)) 

withr]{t,-) real, convex, l.s.c, coercive, and^{t, ) andv real, convex, andC^(H) 
(respectively, l.s.c. in H) in the x variable, defined on H, but not necessarily 
on v. Then we need to assume that ^ and v allow (respectively, l.s.c.) 
extensions git,-) and 4> on the space V . The existence of such extensions is of 
course a strong assumption, see for details and comments upon this matter. 

Moreover, the value function is defined as 

W(t,x)^ inf J(t,x,c), (23) 

ceL^{lt,T]:U) 

Finally, we considered the following (backward) HJB equation associated to the 
problem set in [0,r] x V' 

(vt{t, x) - n{t, B*\/v{t, x)) + {Ax\Vv{t, x)) + g{t, x) = 0, ^^^^ 
for all t in [0, T] and x in D{A) (indeed for all x in V'\ where 

n{t,c) = [h{t,-)n~c). 

Note that Ti is well defined only for p in V , that is a proper subspace of H , to 
which Vv{t,x) (the spatial gradient of w) belongs. 

With such a problem in mind, we then investigate existence and uniqueness 
for the following forward HJB equation 

{(j)t{t,x)^F{t,\/(j){t,x)) - {Ax,V(j>{t,x)) = g{T-t,x), {t,x) e [0,T] x V 
\0(O,x) =^(a;). 

(25) 

Note in fact that such a HJB is the forward version of II24|I if we set 

Fit,p) n{t, B*p) - sup{( - Bc\p)^ - hit, c)}. 
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7.1 Regular data and strong solutions of HJB equations. 

We first treat the case of regular data, from which the notion of strong solution 
originates. 

Assumptions 7.3. 1. A : D{A) C V V is the infinitesimal generator 
of a strongly continuous semigroup {e^^}s>o on V ; 

2. Be L{U,V'); 

3. there exists w > such that \e'^x\^, < Me'^^\x\^,, Vr > 0; 
I Fe y{[0,T] X V), F{t,0) = 0, supt^[o^j,][Fp{t,-)]L < 

5. gey{[0,T]xV'),t^[g^{t,-)l e L\0,T) 

6. ipe So(r); 

7. h{t,-) is convex, lower semi-continuous, dchit,-) is injective for all t e 
[0,T]. 

8. He y{[0,T] X U), nit,0) = 0, anrfsuptg[o,T][Wc(i,-)L < +°°- 
Definition 7.4. Let Assumvtions \ 7. be satisfied. We say that 4> £ C([0, T], C2{V')) 
is a strong solution of ^25\ if there exists a family {4>^}e C C([0, T], C2{V')) such 
that: 

(i) (j)%t,-) e ClipiV) and(j)'{t,-) is convex for allte [0,T]; 0^(0, x) ^ ip{x) 
for all X eV. 

{a) there exist constants ri,r2 > such that 

sup [V0^WL < Ti, sup |V0^(i,O)|y < Ta, Ve > 0; 

tG[0,T] «6[0,T] 

(iii) for all x e F)(A), t 0^(t, is continuously differentiable; 

(iv) (j)" (j), as e ^ 0+, in C{[0,T],C2(y')); 

(v) there exists g^ 6 C{[0,T]:C2{V')) such that, for all t G [0, T] and x £ 
D{A), 

(t)l{t, x) - F{t, \7(l)^t, x)) + {Ax, V(j}'{t, x))v' = ge{T - t, x) 

with ge{t,x) ^ go{t,x), and \gs{s) — go{s)\c2ds 0, as e — > + . 

The main result contained in |E1 is the following. 

Theorem 7.5. Let Assumvtions \7.Si be satisfied. There exists a unique strong 
solution 4> of ^2!^ in the class C([0,T], C2{V')) with the following properties: 

(i) for all X G F){A), 4>{-,x) is Lipschitz continuous; 

(ii) <j){t,-) e Y.o{V'), for allte [0,T]. 

Regarding appHcations to the optimal control problem, in [?] we were able 
to prove what follows. 

Theorem 7.6. Let Assumvtions M .S[ be satisfied, with F(t,p) :— H{t, B*p). Let 
W be the value function of the control problem, and let (j) be the strong solution 
of ^25^ described in Theorem \7.5\ Then 

W{t,x) ^(j){T~t,x), yt e [0,T], Vx e V, 

that is, the value function W of the optimal control problem is the unique strong 
solution of the backward HJB equation \24l . 



17 



7.2 Semicontinuous data and weak solutions of HJB equa- 



We then treat the case of merely semicontinuous data, from which the notion 
of weak solution originates. 

Assumptions 7.7. If K is a convex closed subset ofV, we define 

^ ^k{V') :== {(j) -.V' ^ (-00, +00] : is convex and l.s.c, K C £)('/')} 
where D{(j)) = {x V : (j){x) < +00}, and assume: 

1. C : D{C) C V ^ V is the infinitesimal generator of a strongly continu- 
ous semigroup {e''^}s>o on V' ; 

2. Be L{U,V'); 

3. there exists w > such that \e'^^x\^, < e'^*|a;|^„, Vs > 0; 

4. Fe yi[0,T] X V), F{t,0) = 0, snVt^[o,T][Fp{t,-)]L < 

5. git,-) e T.k{V'), for allte [0,T];g{-,x) l.s.c. andL^{Q,T) for all xe V ; 

6. 

7. h{t,-) is convex, lower semi-continuous, dch{t,-) is injective for all t S 
[0,T]; moreover h{t,c) > a(t)\c\fj + b{t), with a{t) > A{T) > 0, b € 
Li(0,T;R); 

8. He y{[0,T] X U), n{t,0) = 0, anrfsuptg[o,T][Wc(<,-)L < +00. 

Definition 7.8. Let K (Z V be a closed convex set, and let ip G and 

g{t,-) e Sjf for all t in [0,r]. Then (j) ■ [0,T] x V' ^ (— oo,+cxd] is a weak 
solution of (HJB) if: 

(i) 4>{t,-) e Ek, e [0,r]; 

(m) there exist sequences {Lpn}n C Eq; and {gn\ C 3^([0,T] x V'), such that 
Lpn{x)\if{x), gn{t,x) ] g{t,x), Va; G V^', VtG[0,T], as n ^ +00, 
and moreover, if 4>n is the unique strong solution of 



(j)t{t,x) + F{t,y(j){t,x)) - {Ax,V(t){t,x))v' = gn{t,x) {t,x) e [0,T] X V 



Remark 7.9. Since strong solution were proved in flfJ^ to be Lipschtiz with 
respect to the time variable and with respect to the space variable, and the 
weak solution (f) is a sup-envelop of strong solutions (/>„ , then 4> is lower semi- 
continuous in [0,r] X V' . For the same reason 4>n convex in the x variable 
implies that (f) is convex in x as well. 



tions. 




in C{[Q,T],C2{V')), then 



(t>n{t,x)] ^{t,x), y{t,x) e [o,T] X v. 
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Remark 7.10. Note that the role of the convex set K is played in the first 
example by the set 

K ''^^^ clv'{{{xo,xi) : xo > 0}) 

Theorem 7.11. Let Assumntions \7.7i be satisfied. Let also g and h be of the 

following type 

git, x) = e~P'go{x), h{t, c) = e-P%oix) 

. Then the following properties are equivalent: 

(i) there exists a unique weak solution of i25\} : 

(ii) At each (t, x) £ [0, T] x K there exists an admissible control. 
Moreover if (i) or (ii) holds, there exists an optimal pair (c*,y*) and 

(1){T — t,x) — J{t, X, c*). 
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